Systems and Methods for Tracking Moving Objects

ABSTRACT

Systems and methods for tracking moving objects in accordance with embodiments of the invention are disclosed. In one embodiment of the invention, an object tracking system comprises a processor, a communications interface, and a memory configured to store an object tracking application. The object tracking application configures the processor to receive a sequence of images; estimate and subtract background pixel values from pixels in a sequence of images; compute sets of summed intensity values for different per frame pixel offsets from a sequence of images; identify summed intensity values from a set of summed intensity values exceeding a threshold; cluster identified summed intensity values exceeding the threshold corresponding to single moving objects; and identify a location of at least one moving object in an image based on at least one summed intensity value cluster.

CROSS-REFERENCE TO RELATED APPLICATION(S)

The current application claims priority to U.S. Provisional PatentApplication Ser. No. 62/347,979 entitled “Advanced Tracking Camera toDetect Aerial Vehicles Through Synthetic Tracking,” filed Jun. 9, 2016,the disclosure of which is herein incorporated by reference in itsentirety.

STATEMENT OF FEDERALLY SPONSORED RESEARCH

The invention described herein was made in the performance of work undera NASA contract NNN12AA01C, and is subject to the provisions of PublicLaw 96-517 (35 USC 202) in which the Contractor has elected to retaintitle.

FIELD OF THE INVENTION

This invention generally relates to the tracking of moving objects. Moreparticularly, this invention relates to the detection and evaluation ofmoving objects that are not readily visible.

BACKGROUND

Object tracking, or video tracking, is the process of locating andmonitoring an object that is moving using images captured by a camera.This form of tracking can be used in a variety of fields, includingentertainment, security, communications, augmented reality, lawenforcement, military defense, medicine and other scientific endeavors.The object tracking process can be time consuming given the largeamounts of data inherent in video capture. Object tracking aims torelate objects between consecutive frames of a video. Various algorithmsmay be used to analyze video frames for the object's movement betweenframes. Two basic parts of object tracking include detecting the objectand associating detections of the same object over time. Challenges canarise in the case of objects that are fast-moving or change orientation.To handle these situations, motion models may be employed to describepotential changes in the object's appearance or trajectory.

Various object recognition techniques may be employed in the context ofobject tracking to locate and identify objects in an image sequence,using computer vision. Object recognition seeks to mimic the humanability to easily recognize objects despite variables such as differentviewpoints, sizes, rotations, and partial obstruction. Many approacheshave been presented, including appearance-based methods andfeature-based methods. Appearance-based methods use exemplars, ortemplate images, of an object to represent a multitude of appearances ofthe object and perform recognition accordingly. The exemplars mayrepresent the object under varying conditions such as lighting, viewingdirection, size and shape. Feature-based methods aim to match targetobject features with image features, such as surface patches, cornersand/or linear edges.

SUMMARY OF THE INVENTION

Systems and methods for tracking moving objects in accordance withvarious embodiments of the invention are disclosed.

In one embodiment of the invention, an object tracking system comprisesa processor, a communications interface capable of transmitting asequence of images to the processor, and a memory coupled with theprocessor and configured to store an object tracking application.Execution of the object tracking application directs the processor toreceive a sequence of images, wherein at least one moving object isvisible in relation to a background in the sequence of images; estimatepixel background values based on an average of pixel values within asequence of images; subtract background pixel values from pixels in asequence of images; compute sets of summed intensity values fordifferent per frame pixel offsets from a sequence of images, wherein asummed intensity value for a given per frame pixel offset is computed bysumming intensity values of pixels in the images from the sequence ofimages determined using the given per frame pixel offset relative to apixel location in a reference image from the sequence of images;identify summed intensity values from a set of summed intensity valuesexceeding a threshold; cluster identified summed intensity valuesexceeding the threshold corresponding to single moving objects; andidentify a location of at least one moving object in an image based onat least one summed intensity value cluster.

In a further embodiment, the object tracking system further comprises acamera in communication with the processor.

In another embodiment, the object tracking system further comprises aplurality of cameras in communication with the processor.

In a yet further embodiment, execution of the object trackingapplication further directs the processor to perform image stabilizationon a sequence of images.

In another embodiment, the image stabilization is performed bycalculating a Fourier transform for each image in the sequence of imagesin the frequency domain; and multiplying each of the Fourier transformsby a linear phase function, wherein the slope of the linear phasefunction specifies a fractional pixel shift.

In yet another embodiment, the image stabilization is further performedby inverse transforming each of the Fourier transforms back to thespatial domain.

In a further embodiment, the given per frame pixel offset is determinedbased on the fractional pixel shift and a velocity shift.

In still another embodiment, the image stabilization is performed byidentifying at least one camera movement direction based on a set ofadjacent frames of the sequence of images; and degrading the resolutionof each frame of the set of adjacent frames based on the at least onecamera movement direction.

In a still further embodiment, summing intensity values of pixels fromthe sequence of images determined using the given per frame pixel offsetrelative to a pixel location in the reference image is performed bycreating background-subtracted pixels; and summing intensity values ofthe background-subtracted pixels from the sequence of images determinedusing the given per frame pixel offset relative to a pixel location inthe reference image.

In still another embodiment, summing intensity values of pixels from thesequence of images determined using the given per frame pixel offsetrelative to a pixel location in the reference image is performed bysumming the intensity values of the pixels for the given per frame pixeloffset; and subtracting the background pixel values for the summedpixels from the sum of the intensity values of the pixels.

In a yet further embodiment, execution of the object trackingapplication further directs the processor to determine at least one peakin the at least one summed intensity value exceeding a threshold; detecta position of the at least one moving object based on the at least onepeak; and report the position of the at least one moving object to adisplay device in real time.

In yet another embodiment, the at least one peak includes a plurality ofpeaks; and execution of the object tracking application further directsthe processor to determine a per frame sub-pixel offset based on thepixel offsets associated with the plurality of peaks; compute a summedintensity value for the per frame sub-pixel offset by interpolating aplurality of intensity values for the per frame sub-pixel offset, theplurality of interpolated intensity values including an interpolatedintensity value for a plurality of images in the sequence of images; andsumming at least the plurality of interpolated intensity values togenerate the summed intensity value. Execution of the object trackingapplication further directs the processor to estimate a velocity of theat least one moving object based on the per frame sub-pixel offsetassociated with the summed intensity value; and report the velocity ofthe at least one moving object to a display device in real time.

In a further embodiment again, clustering identified summed intensityvalues exceeding the threshold corresponding to single moving objects isperformed by calculating a distance between a first summed intensityvalue and a second summed intensity value in four-dimensional space, thefirst and second summed intensity values being from identified summedintensity values exceeding the threshold; determining whether the firstand second summed intensity values are neighbors based on the calculateddistance; and when the first and second summed intensity values aredetermined to be neighbors, combining the first and second summedintensity values into a summed intensity value cluster.

In another embodiment again, execution of the object trackingapplication further directs the processor to display the location of theat least one moving object in an image from a received sequence ofimages in real time.

In yet another embodiment again, execution of the object trackingapplication further directs the processor to classify the at least onemoving object into an object category.

In still another embodiment, execution of the object trackingapplication further directs the processor to generate an alert based onclassification of the at least one moving object into an objectcategory.

In still yet another embodiment, execution of the object trackingapplication further directs the processor to generate a thumbnail of atleast one moving object; and report a thumbnail of at least one movingobject via a display device in real time.

In a further embodiment, the at least one moving object includes anarticle from the group consisting of an asteroid and an unmanned aerialvehicle (UAV).

An object tracking method, according to another further embodiment ofthe invention, comprises receiving a sequence of images, wherein atleast one moving object is visible in relation to a background in thesequence of images; estimating pixel background values based on anaverage of pixel values within the sequence of images; subtracting thebackground pixel values from pixels in the sequence of images to createbackground-subtracted pixels; computing sets of summed intensity valuesfor different per frame pixel offsets from the background-subtractedpixels, wherein a summed intensity value for a given per frame pixeloffset is computed by summing intensity values of background-subtractedpixels from the sequence of images determined using the given per framepixel offset relative to a pixel location in a reference image from thesequence of images; identifying summed intensity values from the sets ofsummed intensity values exceeding a threshold; clustering the identifiedsummed intensity values exceeding the threshold corresponding to singlemoving objects to form at least one summed intensity value cluster; andidentifying a location of at least one moving object in an image basedon the at least one summed intensity value cluster.

In still another further embodiment of the invention, the objecttracking method further comprises performing image stabilization on thesequence of images.

In a still yet further embodiment, the image stabilization is performedby calculating a Fourier transform for each image in the sequence ofimages in the frequency domain; and multiplying each of the Fouriertransforms by a linear phase function, wherein the slope of the linearphase function specifies a fractional pixel shift.

In still yet another embodiment of the invention, the imagestabilization is further performed by inverse transforming each of theFourier transforms back to the spatial domain.

In a still further embodiment again, the given per frame pixel offset isdetermined based on the fractional pixel shift and a velocity shift.

In still another embodiment again, the image stabilization is performedby identifying at least one camera movement direction based on a set ofadjacent frames of the sequence of images; and degrading the resolutionof each frame of the set of adjacent frames based on the at least onecamera movement direction.

In a yet further embodiment, the object tracking method furthercomprises determining at least one peak in the at least one summedintensity value exceeding a threshold; detecting a position of the atleast one moving object based on the at least one peak; and reportingthe position of the at least one moving object to a display device inreal time.

In another further embodiment, the at least one peak includes aplurality of peaks; and the object tracking method further comprisesdetermining a per frame sub-pixel offset based on the pixel offsetsassociated with the plurality of peaks; computing a summed intensityvalue for the per frame sub-pixel offset by interpolating a plurality ofintensity values for the per frame sub-pixel offset, the plurality ofinterpolated intensity values including an interpolated intensity valuefor a plurality of images in the sequence of images; and summing atleast the plurality of interpolated intensity values to generate thesummed intensity value. The object tracking method further comprisesestimating a velocity of the at least one moving object based on the perframe sub-pixel offset associated with the summed intensity value; andreporting the velocity of the at least one moving object to a displaydevice in real time.

In still another embodiment, clustering identified summed intensityvalues exceeding the threshold corresponding to single moving objects isperformed by calculating a distance between a first summed intensityvalue and a second summed intensity value in four-dimensional space, thefirst and second summed intensity values being from identified summedintensity values exceeding the threshold; determining whether the firstand second summed intensity values are neighbors based on the calculateddistance; and when the first and second summed intensity values aredetermined to be neighbors, combining the first and second summedintensity values into a summed intensity value cluster.

In still yet another embodiment, the object tracking method furthercomprise displaying the location of the at least one moving object in animage from a received sequence of images in real time.

In still another embodiment, the object tracking method furthercomprises classifying the at least one moving object into an objectcategory.

In a further embodiment, the object tracking method further comprisesgenerating an alert based on the classification of the at least onemoving object into the object category.

In another further embodiment, the object tracking method furthercomprises generating a thumbnail of the at least one moving object; andreporting the thumbnail of the at least one moving object to a displaydevice in real time.

In yet another further embodiment, the at least one moving objectincludes an article from the group consisting of an asteroid and anunmanned aerial vehicle (UAV).

An object tracking system, in yet another embodiment of the invention,comprises a processor, a communications interface capable oftransmitting a sequence of images to the processor; and a memory coupledwith the processor and configured to store an object trackingapplication, wherein execution of the object tracking applicationdirects the processor to receive a sequence of images, wherein at leastone moving object is visible in relation to a background in the sequenceof images; perform image stabilization relative to a reference image inthe sequence of images; estimate pixel background values based on anaverage of pixel values within the image stabilized sequence of images;subtract the background pixel values from pixels in the image stabilizedsequence of images to create background-subtracted pixels; compute setsof summed intensity values for different per frame pixel offsets fromthe background-subtracted pixels, wherein a summed intensity value for agiven per frame pixel offset is computed by summing intensity values ofthe background-subtracted pixels from the image stabilized sequence ofimages determined using the given per frame pixel offset relative to apixel location in the reference image; identify summed intensity valuesfrom the sets of summed intensity values exceeding a threshold; clusterthe identified summed intensity values exceeding the thresholdcorresponding to single moving objects to form at least one summedintensity value cluster; and identify a location of at least one movingobject in the reference image based on the at least one summed intensityvalue cluster.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a conceptual illustration of a scene including an objectmoving relative to a background.

FIG. 2 is a diagram of an object tracking system in accordance with anembodiment of the invention.

FIG. 3 is a diagram of a processing system utilized within in an objecttracking system in accordance with an embodiment of the invention.

FIG. 4 is a flow chart illustrating an object tracking method accordingto an embodiment of the invention.

FIG. 5 is a diagram showing a shift-and-add of multiple image framesaccording to an embodiment of the invention.

FIG. 6 shows an image in a sequence of images of a scene containingmoving objects including a UAV against a background.

FIG. 7 shows conceptual illustration of a registration alignment shiftin an image determined as part of an image stabilization process inaccordance with an embodiment of the invention.

FIG. 7A shows an original image captured by a camera.

FIG. 7B shows a version of the image of FIG. 7A, pixelated at theNyquist limit.

FIG. 7C shows a version of the image of FIG. 7A pixelated at the Nyquistlimit and also shifted by 0.245 pixels.

FIG. 8 shows a user interface image with a circled UAV generated inaccordance with an embodiment of the invention.

FIG. 9 shows another user interface image with a circled UAV generatedin accordance with an embodiment of the invention.

FIG. 10 shows a background-subtracted image generated in accordance withan embodiment of the invention.

FIG. 11 conceptually illustrates summed intensity values for images anda chart showing locations of peaks in summed intensity values at a givenpixel location generated in accordance with an embodiment of theinvention.

FIG. 12 illustrates summed intensities of interpolated images addedbased upon a shift incorporating a sub-pixel increment of 1.3 pixels perframe generated in accordance with an embodiment of the invention.

FIG. 13 shows the detection of a drone along with a number of falsepositives in accordance with an embodiment of the invention.

FIG. 14 shows an image of a scene including multiple moving objectsagainst a background.

FIG. 15 shows a background-subtracted image with multiple moving objectsgenerated in accordance with an embodiment of the invention.

FIG. 16 shows an image showing multiple moving objects generated inaccordance with an embodiment of the invention.

FIG. 17 illustrates one configuration of cameras in an object trackingsystem in accordance with an embodiment of the invention.

FIGS. 18-19 illustrate microscopic particle motion tracking inaccordance with an embodiment of the invention.

FIGS. 20-24 illustrate another example of an application of objecttracking methods according to certain embodiments of the invention.

FIG. 20 shows a color image of a scene.

FIG. 21 shows the red component of the image of FIG. 20.

FIG. 22 shows the image of FIG. 21 with background clutter removed.

FIG. 23 shows the image of FIG. 21 with application of an objecttracking method in accordance with an embodiment of the invention.

FIG. 24 shows the image of FIG. 21 with application of an objecttracking method in accordance with another embodiment of the invention.

DETAILED DESCRIPTION

Turning now to the drawings, systems and methods for tracking of movingobjects are illustrated. In many embodiments of the invention, themoving object may not be readily visible to the naked eye, nordetectable with the assistance of a telescope, binoculars, microscope orother viewing device. The tracked object may be any of a plethora ofmoving objects, including but not limited to unmanned aerial vehicles(UAVs, commonly known as drones), asteroids, aircraft, microscopicbodies of interest, and various other traveling articles.

An example of a context in which tracking a moving object may beimportant is that of managing threats posed by UAVs. A non-friendly UAVthat is moving toward a destination may not be readily visible from aremote distance either to the naked eye or via binoculars, cameras andother viewing devices. Once visible, the UAV may already be too close tothe destination for an effective defensive response. Certain embodimentsof the invention may provide a means for detecting and monitoring such aUAV remotely, such that adequate time is provided for a response. It maybe readily appreciated that various other uses exist for manyembodiments of the invention, such as but not limited to redirection ofcommercial UAVs and aircraft safety, the identification of bacteria in awide-field microscope, and/or the tracking of any other objects havingmotion that causes the image of the target to cross pixel boundariesrelative to a background.

An object at a distance, such as but not limited to a UAV of <40 cmdiameter at a distance of >2 km, may appear dim, and thus may require along exposure to be captured in an image. If the object is also moving,it will appear blurry in a long-exposure shot, and its position can bedifficult to ascertain. Attempting to capture the object using a shortexposure, however, typically results in the object appearing too faintfor positive identification. In addition, faint or blurry objects can bedifficult to classify as, for example, a UAV as differentiated from abird.

According to several embodiments of the invention, an object trackingsystem applies a shift-and-add process to a sequence of short high-framerate exposures of a scene, to synthetically render a long-exposure imageof a moving object within the scene. Clustering processes can beutilized to determine the number of moving objects that are presentbased upon the content of the synthetically rendered long-exposureimages. In many instances, a single moving object can appear as multiplehigh intensity pixels within such a synthetically rendered long-exposureimage and a clustering process can be utilized to refine the locationand/or velocity of the moving object. In many embodiments of theinvention, the object tracking process is performed efficiently usingprocesses that can be parallelized on processors such as (but notlimited to) Graphics Processing Units (GPUs). In this way, userinterfaces and/or alerts identifying one or more moving objects can begenerated and presented to one or more users in real time.

The detection, monitoring and reporting of small moving objects may havevarious scientific, medical, commercial, educational and military uses.Although specific examples are discussed above and throughout thepresent specification, it can be readily appreciated that real-timeobject tracking according to various embodiments of the invention may beimplemented in many different fields, and are not limited to thoseparticular examples discussed herein.

Object Tracking Systems

According to a number of embodiments of the invention, an object movingin relation to a background is identified and tracked by an objecttracking system. As an example, FIG. 1 illustrates a scene 100 with abird or moving object 110 moving from position 102 to 104, relative to astationary tree or background 112. This shows a simple example of amoving object 110 moving relative to a background 112. The object's path106, although illustrated with a straight arrow, may be of one or morestraight or curved trajectories. The moving object 110 is shown againsta light background, but may also be flying in front of a lower-contrastbackground 112 such as the tree.

For purposes of this patent application, the scene 100, moving object110 and background 112 as illustrated in FIG. 1 merely exemplify sceneswith moving objects and backgrounds in general, and mentions of thesefigure elements throughout the specification are not intended to limitthe context or type of scene, object or background in any way. Thus,scene 100 can be contemplated to include, but are not limited to,outdoor or indoor backgrounds, and/or any of a variety of moving objectsranging from flying objects to microscopic cells.

According to a number of embodiments of the invention, an objecttracking system 200 such as that illustrated in FIG. 2 receives asequence of images at processing system 210. The images may includeconsecutive exposures of a scene 100. The processing system 210 mayreceive the images via its input/output or communications interface 214,as shown in FIG. 3. These images may be captured and transmitted bycamera 202. Camera 202 may be connected directly to processing system210 as shown in FIG. 2, or may remotely and/or indirectly transmit itsdata to processing system 210 via one of various wired or wirelessmeans. Camera 202 can be one of various types of cameras, including butnot limited to monochrome, RGB, near-IR, and/or arrays of cameras.According to many embodiments of the invention, camera 202 can capturehigh-frame rate exposures of a scene, and may employ one of varioustypes of sensors, such as but not limited to modern scientific CMOS(sCMOS) sensors. In certain embodiments of the invention, a built-innoise reduction feature is disabled in camera 202, so as to preserve thedata in as raw a state as possible for analysis. In many embodiments,imaging parameters are controlled to remain fixed during capture of asequence of images. According to some embodiments of the invention,camera 202 is a color camera, and object tracking methods can be appliedto individual color channels, or to a weighted sum of the differentcolor channels. This may assist in retaining maximum pixel informationand/or in increasing contrast relative to the background. In manyembodiments of the invention, the sequence of images received byprocessing system 210 includes high-frame rate images of scene 100. Theimages may be short-exposure shots, resulting in the moving object 110appearing frozen at various points in its path 106.

As an example if the camera is running at 30 frames/sec, each framecould be exposed for up to 33 msec, less some time for reading out thepixel. If the pixels are saturated, then information about thesubthreshold target could get lost. In that case, the exposure timecould be decreased until the part of the image, in which a target isexpected to be seen, is no longer saturated. Some CMOS sensors can godown to exposures of a few microseconds, and then wait for 33 msec untilit is time to capture the next image. This may also be implemented toreduce motion blur, especially if the camera is on a moving vehicle. Asan example, short exposures may be employed when the camera hassubstantial light-gathering power and the exposure time is reduced toprevent the brightness of the background from driving pixels beyondtheir linear response regime or to saturation, in which case no motionsignal would be observable.

Although the moving object 110 may appear faint or undetectable within asingle frame, according to a number of embodiments of the invention,processing system 210 performs an object tracking process that caninclude a series of shift-and-add computations. These shift-and-addcomputations can be conceptually viewed as a processing in which theimages within the sequence are shifted so that the moving object 110within one image is matched up with the position of the moving object110 within other images. Upon summing the pixels of the shifted images,the moving object 110 may appear more prominently in this syntheticallyrendered long-exposure image. A conceptual illustration of such shiftedand added frames 500, with the object lined up at target track 510, isshown in FIG. 5. Object tracking methods are discussed further in thesections below.

As can readily be appreciated, the location of moving objects in theimages is not known a priori. Accordingly, the shift-and-addcomputations involve adopting a hypothesis that a given pixel within aframe images an object that is moving, and summing pixels across asequence of preceding images corresponding to locations that would beoccupied assuming that the object is moving at a given velocity. Eachpossible velocity (i.e. speed and direction) involves the generation ofa different sum of pixel locations across the sequence of images. Thespecific pixels that are summed can be determined based upon shifts inpixel locations between successive frames in the sequence correspondingto a specific pixel per frame interval velocity. In practice, the shiftsand sum process can result in a single object causing a number of pixelsums corresponding to different velocities. The pixel sum with thehighest intensity can provide the best estimate of the true location andvelocity of an object. In many embodiments, clustering processes can beutilized to determine the number of moving objects that are presentbased upon the observed pixel sums. In several embodiments, movingobjects and their velocities are detected using sums of correspondingpixel locations in the sequence of images at specific velocities thatresult in values that exceed a predetermined threshold. When backgroundsubtraction is applied during the summing process, the correct velocitywill yield a sum in which a high contrast pixel value is reinforced. Thesums of incorrect velocity hypotheses will include many pixels having abackground subtracted intensity (i.e. pixels that do not image themoving object) and so will result in a sum that is lower and less likelyto exceed the threshold. When the sums generated for a given pixellocation do not exceed the threshold at any hypothetical velocity, theprocessing system 210 can determine that the pixel location does notimage a moving object. As can readily be appreciated, the ability toprocess individual pixel locations independently creates significantopportunities to accelerate the shift-and-add computations throughparallelization. Accordingly, processing system 210 in accordance with anumber of embodiments utilizes the parallel processing capabilities ofdevices including (but not limited to) GPUs and/or machine visionprocessors to perform shift-and-add computations during trackingprocesses.

Once the processing system 210 has detected and determined informationregarding at least one moving object, it can transmit information aboutthe moving object via its input/output interface 214 to another device,such as but not limited to a display device 204. Display device 204 maybe connected directly to processing system 210 as shown in FIG. 2, ormay remotely and/or indirectly receive data from processing system 210via one of various wired or wireless means. Display device 204 mayinclude a portable computing device such as but not limited to a tablet,mobile phone, augmented reality visor or heads-up display that includesa client application configured to communicate with a remote server, orit may receive an email alert or other type of message with an imageshowing the moving object 110.

Processing system 210 may be implemented on a single computing device inaccordance with some embodiments of the invention, such as thatillustrated in FIG. 3. Processing system 210 may be a personal computer,a laptop computer, a cloud processing resource, an embedded computingplatform and/or any other computing device with sufficient processingpower for the processes described herein. Processing system 210 mayperform processing locally, or it may partially compute informationlocally, with additional processing being performed on a set of GPUsthat could be located locally or remotely. Processing system 210includes a processor 212, which may refer to one or more devices withinthe computing device that can be configured to perform computations viamachine readable instructions stored within a memory 220 of theprocessing system 210. The memory 220 may contain an object trackingapplication 222 that performs processes such as those described above.The memory 220 may also store various types of data, simultaneously orin succession, such as but not limited to a received sequence of images230 a, registration offsets for image stabilization 230 b,background-subtracted image data 230 c, summed intensity values 230 d, athumbnail of a moving object 230 e, and/or a reference image 230 f.

The processor 212 may include one or more microprocessors (CPUs), one ormore graphics processing units (GPUs), and/or one or more digital signalprocessors (DSPs). According to other embodiments of the invention, theprocessing system 210 may be a system implemented on multiple computers.In some embodiments of the invention, processing system 210 may includean input/output interface 214 that can be utilized to communicate with avariety of devices, including but not limited to a camera 202 and/ordisplay device 204. As can be readily appreciated, a variety of softwarearchitectures can be utilized to implement a computer system 210 inaccordance with several embodiments of the invention. According to manyembodiments of the invention, object tracking systems may compute andreport object information in real time using a graphics processing unit(GPU) and/or parallel processors, thus providing information about themoving object efficiently and allowing sufficient time for a relevantresponse. As an example, processes performed by object tracking systemsmay be parallelized by segmenting the image space and/or the velocitysearch space across multiple processors. The velocity search space,being smaller than the image raster, can be folded into a singledimension for segmentation by a GPU. One of various ways may be used tocache parts of the image and background for optimal processing by aprocessor. Further, many modern GPUs are particularly effective inperforming image processing tasks. As an example, a 4K Ultra HD cameramay produce approximately 1 GB of image data per second. A UAV-detectingvehicle may employ several such cameras, potentially producing acombined terabyte of data approximately every two minutes. Transmissionof such amounts of data from a camera, while inefficient to a remotecomputer, may be practicable in real time to one or more locally orclosely connected GPUs.

While object tracking systems are described above with respect to FIGS.1-3 and 5, other systems may be utilized as appropriate to therequirements of a specific application in accordance with variousembodiments of the invention. Object tracking methods that can beimplemented on any of a variety of object tracking system architecturesin accordance with a number of embodiments of the invention arediscussed further below.

Object Tracking Methods

A method for performing object tracking in accordance with an embodimentof the invention is shown in FIG. 4. In many embodiments of theinvention, a sequence of images is received (402) from one or morecameras. The sequence of images may include consecutive images capturedof a scene, and may include at least one moving object in relation to abackground in the scene. An example of an image in the sequence ofimages is shown in FIG. 6.

In certain embodiments of the invention, image stabilization mayoptionally be performed on the sequence of images, relative to areference image within the sequence of images. Image stabilization mayreduce camera jitter, and may or may not be appropriate depending on thestability and type of camera that captured the images. As an example,image stabilization may not be necessary where cameras are rigidlymounted in a fixed location. However, even rigidly mounted cameras canexperience translations and can benefit from image stabilization. Imagestabilization may be performed on the images using a variety ofprocesses, including (but not limited to) Fourier coefficient-basedimage stabilization. In some embodiments of the invention, onlylow-order components of a Fourier transform are calculated. FIG. 7 showsa conceptual illustration of a registration alignment or offsetcalculated from low-order Fourier components in 0.01 s/frame, orspecifically, the phase of the low-frequency components of the 2DFourier transform of a 256×256 pixel patch of the original grayscaleimage.

According to many embodiments of the invention, a background may beestimated (404) based on an average of pixel values within a sequence ofimages. Drone detection even against a blue sky may require thesubtraction of the background to enhance the detection of the drone, ifthe contrast of the drone against the sky is low. In particular, whenthe variation in the background clutter is larger than the contrast ofthe drone it may be necessary to computationally remove the backgroundbefore attempting to detect the moving object. In certain embodiments ofthe invention, image stabilization (i.e., registering the images) isperformed to align the images, so as to effectively enable furtherprocesses such as background subtraction and shift-and-add calculations,as described below. In the absence of image stabilization, it may not beknown which of the raw pixels sample the same portion of the scene.Accordingly, averaging the pixels would not yield information about thebackground and fixed offsets between frames may not correspond to aconstant velocity. Further, in certain embodiments of the invention,image shifts may be measured at several points across the image, and theamount of shift interpolated depending on the location within the image,as optical aberrations in a wide-field camera system can cause theangular magnification to vary across the field of view.

In cases where a camera is panned or tilted, or if the camera is on amoving vehicle, the images shift as they are being recorded. This imagemotion may often be removed by shifting the image in the computer by acertain integer number of pixels. There are a number of ways to measurethe image shift. One approach is to calculate the cross correlation ofthe two images and look for the peak in the cross correlation.

However, this approach may be imperfect in cases where the image shiftdoes not occur in integer pixel increments. FIG. 7A shows an example ofan object in high resolution, and FIGS. 7B-C show that same objectpixelated with a fractional pixel shift of 0.245 pixels between FIGS. 7Band 7C. Each of the latter two images cannot be created from the other,if limited to integer pixel shifts.

One approach to this issue according to some embodiments of theinvention involves the Shannon sampling theorem, which states that anelectrical signal that is band limited, if sampled at or above theNyquist limit, can perfectly reproduce the original signal. For example,an audio signal limited to 22 Khz, if sampled with an A/D converter at44 Khz, can reproduce the audio signal even on a 1 microsecond timestep. From an information theory point of view, the limited bandwidth ofthe signal indicates a finite amount of information (per unit time), andwhen sampled at or above the Nyquist limit, 100% of the informationcontent can theoretically be measured.

According to certain embodiments of the invention, the Shannon samplingtheorem may be applied on a spatial function in 2D. In the context ofviewing a natural scene, the real world has near infinite informationcontent, in that the scene is the light reflected from a huge number ofatoms. But when viewing the scene through a camera or telescope, theangular resolution is limited by the diffraction of light. Diffractionprovides a natural bandwidth limit to the spatial frequency content ofthe image. If the focal plane is Nyquist sampled, or having more than 2pixels per λ/D, where λ is the highest spatial frequency resolvable bythe optics of the camera, and D is the diameter of the primary lens ofthe camera, the necessary conditions are met for application ofShannon's sampling theorem.

FIGS. 7A-C illustrate such an example. FIG. 7A, the original image, isbandwidth-limited by diffraction. FIG. 7B is a version of the originalimage pixelated at the Nyquist limit. FIG. 7C is a version of theoriginal image pixelated at the Nyquist limit and also shifted by 0.245pixels. Using Shannon's theorem it is possible to reconstruct FIG. 7Cusing FIG. 7B. In many embodiments of the invention, this can be done byfirst calculating the 2D Fast Fourier Transform (FFT) of the image inthe frequency domain, multiplying the result by a 2D linear phasefunction, and then inverse transforming it back to the spatial domain.The slope in x,y of the linear phase function specifies the magnitude ofthe image shift, and that slope can represent an arbitrary fractionalpixel shift.

In performing image stabilization in a sequence of images, according tomany embodiments of the invention, an FFT is performed on each image.This type of processing may be referred to as lossless fractional pixelimage interpolation, and may enable near-perfect background cluttersubtraction from a camera on a moving platform, whether that platform isa car, truck, plane, drone or satellite.

In certain embodiments of the invention, the image stabilization isperformed to calculate the jitter- or motion-induced background offset,which is then added to the shift vector. This implementation may avoidalteration of an image before processing it, and may result in higherefficiency. Further, using merely the phase information from low-orderFourier components of segments of each image may help in the real-timeimplementation. For example, with GPUs, breaking a large image intosmall sub-images and using a single shift vector for the sub-image mayhelp reduce the computation time.

For image jitter resulting from certain types of camera motion, such asslow movement, the lossless fractional pixel image interpolation ofcertain embodiments of the invention described above may be able tofully, or nearly fully, re-register the background, limited by thephoton fluctuations in the detected image. However, in cases of large orsignificant camera jitter from, such as but not limited to, vehicularmovement on an unpaved road, the above solution may be insufficient.Large camera jitter may result in smearing of an image within a singleframe. When camera jitter is large or fast enough to smear the image ina single frame, the smearing may also be different for different frames.

According to several embodiments of the invention, a mechanical solutionto address large camera jitter is to stabilize the camera mechanically,by for example using a 3-axis gimbal. Such gimbals may be mass producedand/or commercially available at reasonable cost, and may be well-suitedto mounting ultra-high definition cameras to ground or flying vehicles.The quality of drone video, for example, may be greatly enhanced whenthe camera is mounted on a servo controlled 3-axis gimbal. The gimbalsmay alleviate camera jitter sufficiently such that when combined withlossless fractional pixel interpolation, the results may approach thatof images captured using a camera fixed to the ground.

In situations where mechanical stabilization of the camera is notavailable, some embodiments of the invention employ a partialcomputational solution. This partial solution estimates the camerajitter by reviewing adjacent frames. For example, if a video is taken at30 hz, within 30 milliseconds the camera is moving in a certaindirection. A few frames later, camera jitter may cause the camera tomove in a different direction. Based on the direction of the jitter foreach frame, each frame can be computationally “degraded” to the samedegraded resolution. Then the background should subtract nearlyperfectly. This is a partial solution because in the end, the resolutionof all the images would be degraded. And while the background cluttercan be subtracted properly, the signal-to-noise ratio (SNR) of anymoving objects within the frames would also be degraded because of thereduced spatial resolution of the images.

According to many embodiments of the invention, the background may thenbe removed (406) from each image in a sequence of images to generate asequence of background-subtracted images. In estimating (404) thebackground, a number of embodiments of the invention average the imagepixels across the sequence of images, resulting in an estimatedbackground independent of the moving object. In several embodiments ofthe invention, removing (406) this background from the sequence ofimages removes the pixels of each frame in which a moving object is notpresent. In certain embodiments of the invention, the background removalprocess may involve computing a rotated or distorted image as indicatedby measurement of the image registration.

While the background subtraction is shown in the flowchart as beingperformed before the shift-and-add (408) computation, according tocertain embodiments of the invention, it could also be performed inconjunction with the shift-and-add process. In the latter case, thebackground calculations can be performed with respect to only thosepixels that are used as inputs to the shift-and-add calculations, thuslimiting the background subtraction to pixel locations that arepotentially relevant to moving objects. In addition, the same imagepatches are used to generate intensity sums and pixel backgroundaverages. Therefore, these operations can be parallelized within a GPU,which could potentially increase memory access efficiency and/or useless memory. In high-contrast settings such as that shown in FIG. 8, thelight-colored UAV circled in the image can be seen against the darkertree background. However, in a low-contrast context such as in FIG. 9where the circled light-colored UAV is flying directly against the sky,the UAV is not readily identifiable. An example image in which thebackground has been subtracted is shown in FIG. 10, where the backgroundremoval results in an isolation of moving objects within the scene. Inthis figure, the portions of the image that were substantiallystationary have been removed, and the moving cars and UAV remain.

In many embodiments of the invention, a shift-and-add process may beperformed upon the sequence of images by computing (408) sets of summedintensity values for different pixel offsets. The pixel offsets mayinclude per frame fixed shifts which, in accordance with someembodiments of the invention, are determined based upon specified and/orassumed velocities of the moving object. Each set of summed intensityvalues for a given pixel offset, relative to a pixel location in aprimary or reference image from the sequence of images, may be computedby applying the given pixel offset to the sequence of images, and thensumming the intensity values of those images. Specifically, pixelswithin the images corresponding to a specified shift per frame aresummed. Since the location of moving objects in the images are likelynot known a priori, a given pixel within a frame may first behypothesized to represent a moving object. In certain embodiments of theinvention, if image registration indicated a fractional frame shift tocompensate for camera motion or image distortion, an interpolation ofthe values of neighboring pixels can be used instead of a specific pixelvalue. As an example, in cases of high background clutter, losslessfractional pixel interpolation may be used to suppress the background ascompletely as possible. In situations where the clutter is less severe,other interpolation techniques may be used. A further assumption may beapplied that the object is moving at a given velocity, and a per-framepixel shift determined based on that velocity. Pixels may then be summedacross a sequence of consecutive images based on per-frame shifts inpixel locations between successive frames. Different sums of pixellocations across the sequence of images arise for each of the differentper frame shifts corresponding to various possible velocities. When asum is determined to exceed a threshold value, the presence of a movingobject is confirmed and its resulting velocity estimated based on theper frame shift upon which the pixels were summed. When backgroundsubtraction is applied to images shifted according to a per frame shiftbased on the correct velocity, the result reinforces a high contrastpixel value. The sums of values according to per frame shifts based uponincorrectly hypothesized velocities, for pixels that do not image themoving object, will result in a lower sum that is less likely to exceedthe threshold.

The result of performing shift-and-add calculations at each pixellocation within a region of an image with respect to each of a number ofdifferent velocities is conceptually illustrated in FIG. 11. Each imagein FIG. 11 corresponds to the summed intensities at a given pixellocation, generated using a specific per frame shift relative to thepixel location to select pixels from a sequence of images, and thensumming the selected pixels. The images correspond to different perframe integer pixel shifts in the x and/or y direction. While manyembodiments simply perform this processing in memory, the images serveto conceptually illustrate the peaks in intensity that occur in thelocation of a moving object. As can readily be appreciated from theintensity sums generated assuming different per frame shifts, increasesin intensity are observed in all images in the location of a movingobject. The extent to which the per frame pixel offset corresponds tothe velocity vector of the moving object determines the magnitude of theintensity peak. The images that are generated using per frame pixeloffsets that most closely approximate the moving object's actualvelocity have the highest intensities, and the other images tend to havepeaks in intensity that are spread over a larger number of pixels. Thedirection of the spread in intensity can provide information that isutilized by the process to estimate the actual velocity of the movingobject. As is discussed further below, the integer pixel shifts thatyield the largest intensity peaks can be utilized to recompute intensitysums by interpolating pixels in the sequence of images corresponding tovalues at sub-pixel shifts. In this way, the accuracy of the estimatedlocation of the moving object and/or its estimated velocity can berefined.

Since the presence of a moving object causes a change in the intensityof a pixel imaging a moving object in a particular frame is relative tothe average or background value of the pixel that is typically observedin the pixel location in the absence of the moving object. When pixelvalues are summed, those pixels imaging a moving object will appear ashigh-contrast pixels in relation to the other background-subtractedpixels in the image. According to several embodiments of the invention,summed intensity values from the sets of summed intensity valuesexceeding a threshold may be identified (410). A location of at leastone moving object in the reference image may also be identified based ona summed intensity value from a set of summed intensity values exceedinga threshold. The threshold may be pre-determined and/or fixed. Thesummed intensity values exceeding a threshold may also be determinedadaptively based upon the magnitude of a peak summed intensity valuerelative to the summed intensity values of surrounding pixel locationsSummed intensity values with different per frame pixel offsets are shownin FIG. 11, with the largest peaks obtained for per frame pixel offsetsof (dx, dy)=(−1,0) and (dx, dy)=(−2,0).

The high summed intensity values can thus include more than one peak,from which the moving object's position and velocity may be estimated,in accordance with some embodiments of the invention. In certainembodiments of the invention, the position and/or velocity can berefined at a sub-pixel level by determining a sub-pixel shift based onthe pixel offsets associated with the plurality of peaks. The sub-pixelshift may be determined by weighting the peak pixel offsets, or byassuming per frame shifts with sub-pixel components. Interpolation canthen be used to generate intensity values at the sub-pixel locations.These values can then be summed to generate a summed intensity value forthe sub-pixel shift. The result should yield the highest intensity peak,with its value estimating a refined velocity of the moving object.

The example in FIG. 12 shows a per frame shift of (−1.3, 0) which isbetween the two highest peaks of (−1, 0) and (−2, 0). The selection of(−1.3, 0) could be based upon sampling a set of sub-pixel shifts withina range between of (−1, 0) to (−2, 0) or could be determined in a singlecalculation by determining a sub-pixel shift based upon a weighting ofthe intensity peaks at (−1, 0) and (−2, 0). A moving object can bedetermined to exist in the set of images when the absolute value(because the moving object could be darker or brighter than thebackground) of the highest peak exceeds a threshold value, or when aresampled thumbnail at the interpolated velocity exceeds a thresholdvalue. The threshold value can be determined by an operator or becomputed from statistics of the surrounding imagery.

In performing multivector shift-and-add processes, at the correctlyhypothesized velocity, the photons may add up to present a bright sharpimage of the moving object. However, if the object is bright, a slightlyincorrect velocity may still appear as a slightly less bright and/orslightly blurred image, because the velocity is not accurate but stillwell above the threshold. This may occur for multiple differentvelocities close to the correct velocity. In order to distinguishbetween a single object and numerous distinct objects, in severalembodiments of the invention, a clustering (412) method according tocertain embodiments of the invention may be used, and a location of atleast one moving object identified (414) in an image based on a summedintensity value cluster.

In certain embodiments of the invention, the final position and/orvelocity of the moving object may be discerned by calculating themultivector shift-and-add on a grid of velocities. This grid may havetwo parameters, the spacing of velocities as grid points and the extentof the grid. The extent of the grid may be set by the user. As anexample and not by way of limitation, a drone moving 100 mph at adistance of 1 mile is moving at an angular rate of 28 milliradians/sec.If the camera has a 1 radian FOV across 2000 pixels (standard HDTVcamera) and running at 30 hz, this max motion is 2 pixels/frame. Theshift-and-add process may then be performed over the grid of velocities.The real velocity may not be exactly one of these grid points. Thespacing of the grid points may be chosen so that at the nearest gridpoint, the SNR is only degraded by a certain percentage, such as 10˜20%.

After the multi-vector shift-and-add process finds a cluster of pointsin 4D above the detection threshold, the final determination may be madeof the best fit position and velocity. The final step may involvetesting fraction pixel velocities that results in the most compactsynthetic image. This may be performed with a gradient search, which canbe quite fast. A standard non-linear least squares fitting routine maybe used to find the position and velocity that results in a minimumvariance fit to the data.

In the context of imaging moving objects, an object may straddle, forexample, two pixels or corners of four pixels, and thus light upmultiple pixels. In many applications, it may be useful to performautomatic identification of the objects, as given the enormous volume ofvideo data available with very low cost cameras, it may be infeasiblefor a human to review each video feed in detail.

In an object tracking process for space debris, as an example, numerousvelocities may be examined. Given a multiple-image data set, applying anobject tracking application according to an embodiment of the inventionto each of the velocities results in a number of synthetic images. As asingle bright moving object may result in many above-threshold pixels,the output of the shift-and-add process may produce numerous pixels thatare above threshold. At this point, it may be unclear as to whetherthere exist a significant number of moving objects, a single movingobject, or a number in between.

A clustering method in accordance with many embodiments of the inventioncan sort through the clutter and arrive at the correct number of movingobjects in the scene. In a set of input data including multiple images,each image may comprise numerous pixels. The multiple-image input may beconsidered to be a single data cube as a function of x, y, and time.Likewise, the output of synthetic images corresponding to the velocityvectors may instead be considered to be a single four-dimensional (4D)data cube.

In several embodiments of the invention, the clustering methodidentifies all above-threshold pixels in the 4D data cube and calculatesthe 4D distances between those points. When the distance between twopixels in this 4D space represents a neighboring pixel, the two pixelsare collected and identified as a single object. In 1D, a pixel has twoneighbors. In 2D, every pixel has 8 neighbors; in 3D, 26 neighbors; andin 4D, 80 neighbors. As an example, if the distance between adjacentpixels in 1D is designated to be 1 pixel, then in 2D a corner pixel hasa distance of 1.4 pixels from the center pixel. Similarly, in 3D and 4Dthe corner pixel is 1.7 pixels and 2.0 pixels from the center pixel.When applying the clustering method in 4D, according to certainembodiments of the invention, any two pixels that are 2 or fewer unitsapart may be clustered. Alternatively, a neighboring pixel can bedesignated as being 2 pixels distant.

Neighboring pixels may then be combined into object clusters. Thus, forexample, if there are 10,000 points above threshold but only 5 distinctobjects, the output of the clustering method would be the 5 points thatrepresent the highest summed intensities in each of the 5 objectclusters.

According to some embodiments of the invention, a fractional pixelinterpolation operation may be used find the final best fit position andvelocity after the clustering operation. The output of the clusteringmethod can be sent to a non-linear least squares fitting routine thatperforms fractional pixel shift-and-add to find the position andvelocity of the object. This process may be performed with a precisionthat is limited no longer by pixelation effects, but only the photonnoise in the images.

One phenomenon in the detection of UAV versus the detection of objectsin space is illustrated in FIG. 13. After the multishift/add operation,a threshold may be set, above which are considered potential movingobjects and below which is considered noise in the background. FIG. 13shows that the threshold that allows the detection of a drone may alsoproduce a significant number of false positives. The false positives inthis example result from trees in the image because their leaves rustlein the wind. The motion of the leaves cause each frame in the video tobe slightly different. By chance some of these fluctuations will line upto mimic linear motion over a short period of time. One solutionaccording to some embodiments of the invention comes from realizing thatthe noise in different parts of the image is different. Instead ofsetting the threshold at a certain flux level, it may be set at a fixedsignal to noise ratio. The area with leaves rustling would have highnoise and would need a higher flux level to trigger a claim ofdetection. The noise level can be measured by comparing the differencesin the successive frames of the video.

Further, in certain embodiments of the invention, a thumbnail of themoving object may be generated and displayed or reported to a displaydevice in real time. Some embodiments of the invention may also computeand report the object's size, position and/or velocity. Informationregarding the moving object may be reported and/or displayed relative toa reference image. The thumbnail, in combination with other informationsuch as velocity and trajectory, can also be utilized to classify theobject as a certain type of object, or into a certain category. Certainembodiments of the invention generate an alert based on theclassification of the moving object. The alert may be visual, audio,generated locally and/or transmitted and/or broadcast. Examples ofclassifications include, but are not limited to, a UAV, asteroid,missile, quadcopter, helicopter, plane, bird, road vehicle, football,Frisbee and other categories of varying specificity.

In certain embodiments of the invention, a “filmstrip”, or a series ofthumbnails, which could be a time series of original images or resampledthumbnails created by interpolating a subset of the total set ofcaptured images, can be used to assess whether there is a periodicity inthe motion, as in the flapping of a bird or the waving of a leaf in thebreeze. Successive detections can be used to infer motion over time,where motion over time may be used to identify a class of objects, suchas but not limited to footballs following parabolic arcs, rocketstraveling in more or less straight lines, waving flags and trees movingback and forth periodically, and/or UAVs and airplanes moving withconstant velocity. As an example, the amount of light a flapping birdwing reflects to a camera may vary periodically, so the brightness ofthe thumbnail image (with background removed) over time can identifyperiodic motion. Although flapping-bird UAVs exist, a robot bird tendsto fly in a straight line while a real bird makes constant coursecorrections to the path it takes over several seconds. This phenomenoncould be used to classify whether the object is a bird or a bird-bot.

A further example of a scene to which object tracking methods areapplied according to certain embodiments of the invention is shown inFIGS. 14-16. FIG. 14 shows a frame of multiple moving objects in thesky. FIG. 15 shows the scene with the background mostly subtracted out.After applying a threshold, moving objects identified to be above thethreshold are automatically identified as shown in FIG. 16. The movingobjects in this image appear as the locations of the intensity summedpixels from a sequence of images that have an intensity sum at aspecific per frame pixel offset that exceeds a threshold. According tocertain embodiments of the invention, the points of the moving objectsshown are potentially moving at different velocities, so the pixelshifts that yielded a sum that is above the threshold at each locationmay be different.

In accordance with many embodiments of the invention, the processesdescribed above in the present application may be applied in a number offields and contexts. In many embodiments of the invention, the objecttracking can be performed in real time, by employing, for example GPUsand/or parallel processors as discussed in the above section. Thereal-time reporting of information may provide sufficient time for arelevant response. For example, according to certain embodiments of theinvention, an object tracking system may detect and track a UAVindicating a threat, and provide a report and/or alert with sufficienttime (e.g., at least ten seconds) to respond. The image and motionanalytics provided by the object tracking system, in some embodiments ofthe invention, may be fed into a threat-assessment pipeline.

While object tracking methods are described above with respect to FIGS.4-16, other methods may be utilized appropriate to the requirements of aspecific application in accordance with various embodiments of theinvention. In addition, the processes described above can be performedin different orders as appropriate to the requirements of a specificapplication, system hardware and other factors. Camera configurationsfor object tracking in accordance with a number of embodiments of theinvention are discussed further below.

Object Tracking Cameras and Configurations

Several embodiments of the invention may employ a plurality of camerasto provide a 4-pi steradians full spherical field of view, 360-degree,or wide coverage of a scene of interest, and/or stereo targeting of amoving object. FIG. 17 illustrates one specific configuration of cameras202 a-h, in which four cameras are arranged on each of two poles 203a-b. This configuration may enable 360-degree capture of the surroundingarea. In addition, since moving object 110 a may be seen by two or morecameras at separate locations, the captured image data may be used toprovide three-dimensional positional and/or velocity vector informationof the object. As examples, for drone detection, individual cameras withlarge fields of view, four cameras may cover 360 degrees. In using spacesurveillance cameras with smaller fields of view, in certain embodimentsof the invention, multiple cameras may be combined to extend the fieldof view.

Cameras according to embodiments of the invention may be disposed uponvehicles, helmets, and various other devices as appropriate to specificapplications. In accordance with some embodiments of the invention,computers implementing object tracking methods may be locally connectedto the cameras on each device. Cameras and/or object tracking systemsaccording to certain embodiments of the invention may be arranged tocommunicate with each other, to provide alerts, analytical coordination,and/or other cooperative functions.

Many embodiments of the invention can be performed in a multiviewenvironment, in which object detections in one field of view can bevalidated with object detections in another field of view, and in whichrange can be potentially estimated. Object detections in the field ofone camera may also be used to accelerate the search for moving objectsin another camera. For example, if a known object exists within thefield of view of one camera, a search may be performed along an epipolarline, or within a band around the epipolar line to account for alignmenterrors and/or errors in the position estimate from the first camera, tolook for moving objects. Using epipolar geometry in such a manner, thelocation of the moving object can be confirmed, as the second viewpointvalidates the first viewpoint, and the distance along the epipolar lineprovides an estimate of the distance of the moving object.

In addition, in certain embodiments of the invention, two velocityvectors may be calculated for the moving object and can be used inconjunction with epipolar geometry to further refine both the distanceand the true velocity estimate for the moving object. Specifically,velocity vectors perpendicular to the optical axis of each camera may bedetermined, and changes in distance over time can yield estimates ofvelocity in a direction parallel to the optical axis of at least one ofthe cameras. Given that the differences in the velocity vectors in eachviewpoint are related to distance, the two velocity vectors can be usedwith the depth information to refine both distance and velocityestimates. Further, in many embodiments of the invention, thumbnailsgenerated for the object based on the fields of both cameras may beprovided to a classifier to increase the likelihood of correctclassification.

While cameras and configurations for object tracking are described abovewith respect to FIG. 17, it can be readily appreciated that varioustypes, numbers and configurations of cameras may be used in a givenobject tracking system, as appropriate for the requirements of itsspecific application in accordance with various embodiments of theinvention. Applications of object tracking systems in accordance with anumber of embodiments of the invention are discussed further below.

Applications of Object Tracking Systems

The object tracking systems and methods described throughout the presentspecification may be applied in a number of fields and contexts. In manyembodiments of the invention, object tracking can be performed in realtime, with information regarding a moving object being reported insufficient time to allow for a relevant response. For example, accordingto certain embodiments of the invention, an object tracking system maydetect and track an UAV indicating a threat, and provide a report and/oralert with sufficient time (e.g. at least ten seconds) to respond. Theimage and motion analytics provided by the object tracking system, insome embodiments of the invention, may be fed into a threat-assessmentpipeline. In some embodiments of the invention, a recommendation for anappropriate response may be provided, and may involve, for example, thelaunch of a countermeasure to oppose, neutralize and/or retaliate withregard to the moving object including (but not limited to) launching aprojectile, launching a net, and/or launching a counterstrike UAV.

While much of the discussion above is related to detection and trackingof UAVs, systems and methods in accordance with many embodiments of theinvention can be utilized to track moving objects in a variety ofcontexts. The real-time object tracking of several embodiments of theinvention can be used in many scientific, medical, commercial,educational, military and other contexts. In the medical field, objecttracking systems may be used to identify bacteria in a low-resolutionmicroscope. In certain embodiments of the invention, object tracking canbe applied to human blood under a microscope to detect bacteria.High-resolution optics may then be used to determine the type ofbacteria and suggest treatment options.

As an example, FIGS. 18-19 illustrate the tracking of particle motion inmicrobial life from pond water, under a digital holographic microscope.In a low resolution imaging system, the microorganisms may be moving andas a result they are being sampled differently in each consecutiveimage. According to some embodiments of the invention, an objecttracking method may be used to track the microorganisms and crop imagepatches that contain a single microorganism. The image patches may thenbe aligned using an image stabilization process. This may generate aseries of images of a single organism that sample the organism at slightsub-pixel offsets. This property of sampling a scene with sub-pixeloffsets can be exploited to perform super-resolution processing.Super-resolution processes enable synthesis of images that are higherresolution than the images captured by the microscope. This techniquecould also be utilized in a UAV context, especially in circumstances inwhich the cameras are fixed.

In some embodiments of the invention, in the holographic microscope, theraw data is first processed into a three-dimensional data cube of pixels(or voxels), and over time the microscope produces a time series ofthese data cubes. In certain embodiments of the invention, the objecttracking methods may be applied to this data set, when the images of thecreatures to be detected are larger than a voxel, by slicing the datacubes along one dimension to make a stack of 2D images, combiningmatching slices from successive data cubes to make a 2D movie, andrunning the object tracking processes on that. Once the velocity trackof a moving object is determined, super-resolution processing entailsshifting and adding successive frames, and applying clustering methodssuch as those described above. Stitching together position andvelocities, similarly as that described above with regard to clusteringmethods, it may be determined, for example, that the long track ofcircles labeled “In context” near the middle of the raw video of FIG.18, represents a life form, whereas the “software errors” at the edgeare false positives. The software errors may result from noise in theimage triggering a detection that looks like a moving object in a singleset of frames, but when overlayed on a longer time series, theydisappear and reappear possibly without even following a random walk. Itmay be concluded that they do not actually represent a life form. Themotion pattern discrimination in a microscope may be different than for,as an example, a UAV. Glints and image noise typically do not appear tohave correlated motion of time, whereas dust in a microscope moves in arandom walk because of Brownian motion, and organisms that propelthemselves move, at least for some time scale, in relatively straightlines which are statistically highly unlikely to have arisen from arandom walk.

This analysis may be performed on a single slice of the data cube, upontwo spatial dimensions and one time dimension. However, since theholographic microscope may have 3+1 dimensions (3 spatial and 1 time),the shift and add processes described herein can be extended to searchover all velocities in 3 dimensions. This may require about 10 times asmuch processing power as with 2 dimensions, but it can be parallelized.In some implementations of the microscope, fluid is pumped constantly(and slowly) through the observing area, so the motion jitter may becomputed in a subsample of a few image planes, as in the 2D problem,before distributing the imagery to GPUs for processing. Likewise, thethumbnail interpolation and classification processes would all beperformed in 3D instead of 2D.

In certain embodiments of the invention, a 3D version of object trackingmethods as discussed above may be implemented. A 6D volume may beemployed to determine 3D positions and velocities.

Object tracking systems and methods in accordance with many embodimentsof the invention may also be applied to detecting objects in space. Indetecting objects in space, such as but not limited to satellites orasteroids, it may be important not only to detect them but also to atleast roughly measure their orbits. While approximately 1200 newnear-Earth asteroids are detected every year, the majority of these aresubsequently lost. Orbit measurement is similarly applicable in themonitoring of spacecraft and space debris in Earth's orbit.

When a new moving object in space is detected, a single detection isoften insufficient because that observation cannot be linked with prioror subsequent detections of the same object. Object tracking processesin accordance with many embodiments of the invention as described abovecan be beneficial in tracking space objects, given the measurement ofboth position and velocity in these processes.

Using a typical 1 m telescope, an asteroid may require an observationperiod of 20-30 days in order for the orbit to have sufficient precisionto enable linking of that set of observations to future detections ofthe asteroid. Where a detection denotes a single instance where theobject is detected, a cataloged observation set denotes a series ofobservations that allow a crude or estimated orbit to be derived thatenables subsequent detections to be linked to this initial orbit. Forearth-orbiting satellites near geostationary orbit (GEO) altitude, therequired observation period is approximately 2 hrs. However, manyasteroids are not detectable over a time span of 20 or 30 days. Theasteroid may, for example, move from the nighttime sky to the daytimesky.

Many embodiments of the invention enable much smaller telescopes todetect small, fast moving objects. As an example, 15 cm telescopes usedin conjunction with object tracking methods, may render the samesensitivity as 1 m telescopes alone, and lower costs significantly. Byitself, the sensitivity of a camera may depend only on the size of itsentrance aperture. As an example, the LSST telescope is a facility withan ˜8 m telescope, a 3.4 gigpixel focal plane and 9 sqdeg field of view.But the sensitivity of the large aperture may be duplicated with a muchsmaller telescope that integrates much longer than 15 sec per exposure,when used with object tracking methods in accordance with manyembodiments of the invention. This advantage may be considered to arisefrom the cost scaling for telescopes, with large telescope costs risingsignificantly. With the use of object tracking methods, similarperformance may be obtained using an array of smaller telescopes withthe same collecting area. As a particular example, an array of 50 cmtelescopes with the same equivalent area would be less expensive than asingle large telescope by about a factor of 4.

As applied to asteroid and satellite detection, this lower coststructure can enable a new structure for initial orbit determination.That is, the use of two low cost telescopes separated by some distance,can render a 3D, rather than 2D, measurement of the position of anobject in space. Two telescopes viewing the same object in space allowfor a parallax (or triangulation) distance measurement. With 3Dmeasurement, an asteroid's orbit can be determined possibly with 2measurements 2 days apart, in contrast to requiring a series ofmeasurements separated by 20-30 days. A similar concept applies tosatellites in Earth's orbit, although on a different timescale.

FIGS. 20-24 illustrate another example of an application of objecttracking methods according to certain embodiments of the invention. FIG.20 shows a color image of a scene. FIG. 21 shows the red component ofthe image of FIG. 20. In certain embodiments of the invention, the redchannel may be analyzed for higher contrast because much of the scene isblue because of the sky. FIG. 22 shows the image of FIG. 21 withbackground clutter removed, but without use of the object trackingmethods according to many embodiments of the invention. Here, the treesare not perfectly removed because the leaves are blowing in the wind. Inactuality, there were two planes flying left to right above the trees,one moving at ˜10 pix/sec and the other at the other ˜20 pix/sec. FIG.23 shows the image of FIG. 21 with application of object trackingmethods according to certain embodiments of the invention. Here, oneairplane can be seen (flying at ˜20 pix/sec). In FIG. 24, objecttracking methods are applied to the other plane flying at ˜10 pix/secand behind smog. The flux from the plane is 50 times lower than the skybackground.

Although certain embodiments of the invention have been described abovewith respect to FIGS. 18-24 and other specific applications, it may becontemplated that one or more object tracking systems and methods asdiscussed above may be utilized in numerous other applications inaccordance with various embodiments of the invention.

CONCLUSION

Although the present invention has been described in certain specificaspects, many additional modifications and variations would be apparentto those skilled in the art. It is therefore to be understood that thepresent invention can be practiced otherwise than specifically describedwithout departing from the scope and spirit of the present invention.Thus, embodiments of the present invention should be considered in allrespects as illustrative and not restrictive. Accordingly, the scope ofthe invention should be determined not by the embodiments illustrated,but by the appended claims and their equivalents.

What is claimed is:
 1. An object tracking system, comprising: aprocessor; a communications interface capable of transmitting a sequenceof images to the processor; and a memory coupled with the processor andconfigured to store an object tracking application, wherein execution ofthe object tracking application directs the processor to: receive asequence of images, wherein at least one moving object is visible inrelation to a background in the sequence of images; estimate pixelbackground values based on an average of pixel values within a sequenceof images; subtract background pixel values from pixels in a sequence ofimages; compute sets of summed intensity values for different per framepixel offsets from a sequence of images, wherein a summed intensityvalue for a given per frame pixel offset is computed by summingintensity values of pixels in the images from the sequence of imagesdetermined using the given per frame pixel offset relative to a pixellocation in a reference image from the sequence of images; identifysummed intensity values from a set of summed intensity values exceedinga threshold; cluster identified summed intensity values exceeding thethreshold corresponding to single moving objects; and identify alocation of at least one moving object in an image based on at least onesummed intensity value cluster.
 2. The object tracking system of claim1, further comprising: a camera in communication with the processor. 3.The object tracking system of claim 1, further comprising: a pluralityof cameras in communication with the processor.
 4. The object trackingsystem of claim 1, wherein execution of the object tracking applicationfurther directs the processor to: perform image stabilization on asequence of images.
 5. The object tracking system of claim 4, whereinthe image stabilization is performed by: calculating a Fourier transformfor each image in the sequence of images in the frequency domain; andmultiplying each of the Fourier transforms by a linear phase function,wherein the slope of the linear phase function specifies a fractionalpixel shift.
 6. The object tracking system of claim 5, wherein the imagestabilization is further performed by: inverse transforming each of theFourier transforms back to the spatial domain.
 7. The object trackingsystem of claim 5, wherein the given per frame pixel offset isdetermined based on the fractional pixel shift and a velocity shift. 8.The object tracking system of claim 4, wherein the image stabilizationis performed by: identifying at least one camera movement directionbased on a set of adjacent frames of the sequence of images; anddegrading the resolution of each frame of the set of adjacent framesbased on the at least one camera movement direction.
 9. The objecttracking system of claim 1, wherein summing intensity values of pixelsfrom the sequence of images determined using the given per frame pixeloffset relative to a pixel location in the reference image is performedby: creating background-subtracted pixels; and summing intensity valuesof the background-subtracted pixels from the sequence of imagesdetermined using the given per frame pixel offset relative to a pixellocation in the reference image.
 10. The object tracking system of claim1, wherein summing intensity values of pixels from the sequence ofimages determined using the given per frame pixel offset relative to apixel location in the reference image is performed by: summing theintensity values of the pixels for the given per frame pixel offset; andsubtracting the background pixel values for the summed pixels from thesum of the intensity values of the pixels.
 11. The object trackingsystem of claim 1, wherein execution of the object tracking applicationfurther directs the processor to: determine at least one peak in the atleast one summed intensity value exceeding a threshold; detect aposition of the at least one moving object based on the at least onepeak; and report the position of the at least one moving object to adisplay device in real time.
 12. The object tracking system of claim 11,wherein: the at least one peak includes a plurality of peaks; andexecution of the object tracking application further directs theprocessor to: determine a per frame sub-pixel offset based on the pixeloffsets associated with the plurality of peaks; compute a summedintensity value for the per frame sub-pixel offset by: interpolating aplurality of intensity values for the per frame sub-pixel offset, theplurality of interpolated intensity values including an interpolatedintensity value for a plurality of images in the sequence of images; andsumming at least the plurality of interpolated intensity values togenerate the summed intensity value; estimate a velocity of the at leastone moving object based on the per frame sub-pixel offset associatedwith the summed intensity value; and report the velocity of the at leastone moving object to a display device in real time.
 13. The objecttracking system of claim 1, wherein clustering identified summedintensity values exceeding the threshold corresponding to single movingobjects is performed by: calculating a distance between a first summedintensity value and a second summed intensity value in four-dimensionalspace, the first and second summed intensity values being fromidentified summed intensity values exceeding the threshold; determiningwhether the first and second summed intensity values are neighbors basedon the calculated distance; and when the first and second summedintensity values are determined to be neighbors, combining the first andsecond summed intensity values into a summed intensity value cluster.14. The object tracking system of claim 1, wherein execution of theobject tracking application further directs the processor to display thelocation of the at least one moving object in an image from a receivedsequence of images in real time.
 15. The object tracking system of claim1, wherein execution of the object tracking application further directsthe processor to classify the at least one moving object into an objectcategory.
 16. The object tracking system of claim 1, wherein executionof the object tracking application further directs the processor togenerate an alert based on classification of the at least one movingobject into an object category.
 17. The object tracking system of claim1, wherein execution of the object tracking application further directsthe processor to: generate a thumbnail of at least one moving object;and report a thumbnail of at least one moving object via a displaydevice in real time.
 18. The object tracking system of claim 1, whereinthe at least one moving object includes an article from the groupconsisting of an asteroid and an unmanned aerial vehicle (UAV).
 19. Anobject tracking method, comprising: receiving a sequence of images,wherein at least one moving object is visible in relation to abackground in the sequence of images; estimating pixel background valuesbased on an average of pixel values within the sequence of images;subtracting the background pixel values from pixels in the sequence ofimages to create background-subtracted pixels; computing sets of summedintensity values for different per frame pixel offsets from thebackground-subtracted pixels, wherein a summed intensity value for agiven per frame pixel offset is computed by summing intensity values ofbackground-subtracted pixels from the sequence of images determinedusing the given per frame pixel offset relative to a pixel location in areference image from the sequence of images; identifying summedintensity values from the sets of summed intensity values exceeding athreshold; clustering the identified summed intensity values exceedingthe threshold corresponding to single moving objects to form at leastone summed intensity value cluster; and identifying a location of atleast one moving object in an image based on the at least one summedintensity value cluster.
 20. The object tracking method of claim 19,further comprising: performing image stabilization on the sequence ofimages.
 21. The object tracking method of claim 20, wherein the imagestabilization is performed by: calculating a Fourier transform for eachimage in the sequence of images in the frequency domain; and multiplyingeach of the Fourier transforms by a linear phase function, wherein theslope of the linear phase function specifies a fractional pixel shift.22. The object tracking method of claim 21, wherein the imagestabilization is further performed by: inverse transforming each of theFourier transforms back to the spatial domain.
 23. The object trackingmethod of claim 21, wherein the given per frame pixel offset isdetermined based on the fractional pixel shift and a velocity shift. 24.The object tracking method of claim 20, wherein the image stabilizationis performed by: identifying at least one camera movement directionbased on a set of adjacent frames of the sequence of images; anddegrading the resolution of each frame of the set of adjacent framesbased on the at least one camera movement direction.
 25. The objecttracking method of claim 19, further comprising: determining at leastone peak in the at least one summed intensity value exceeding athreshold; detecting a position of the at least one moving object basedon the at least one peak; and reporting the position of the at least onemoving object to a display device in real time.
 26. The object trackingmethod of claim 25, wherein: the at least one peak includes a pluralityof peaks; and the method further comprises: determining a per framesub-pixel offset based on the pixel offsets associated with theplurality of peaks; computing a summed intensity value for the per framesub-pixel offset by: interpolating a plurality of intensity values forthe per frame sub-pixel offset, the plurality of interpolated intensityvalues including an interpolated intensity value for a plurality ofimages in the sequence of images; and summing at least the plurality ofinterpolated intensity values to generate the summed intensity value;estimating a velocity of the at least one moving object based on the perframe sub-pixel offset associated with the summed intensity value; andreporting the velocity of the at least one moving object to a displaydevice in real time.
 27. The object tracking method of claim 19, whereinclustering identified summed intensity values exceeding the thresholdcorresponding to single moving objects is performed by: calculating adistance between a first summed intensity value and a second summedintensity value in four-dimensional space, the first and second summedintensity values being from identified summed intensity values exceedingthe threshold; determining whether the first and second summed intensityvalues are neighbors based on the calculated distance; and when thefirst and second summed intensity values are determined to be neighbors,combining the first and second summed intensity values into a summedintensity value cluster.
 28. The object tracking method of claim 19,further comprising: displaying the location of the at least one movingobject in an image from a received sequence of images in real time. 29.The object tracking method of claim 19, further comprising: classifyingthe at least one moving object into an object category.
 30. The objecttracking method of claim 29, further comprising: generating an alertbased on the classification of the at least one moving object into theobject category.
 31. The object tracking method of claim 19, furthercomprising: generating a thumbnail of the at least one moving object;and reporting the thumbnail of the at least one moving object to adisplay device in real time.
 32. The object tracking method of claim 19,wherein the at least one moving object includes an article from thegroup consisting of an asteroid and an unmanned aerial vehicle (UAV).33. An object tracking system, comprising: a processor; a communicationsinterface capable of transmitting a sequence of images to the processor;and a memory coupled with the processor and configured to store anobject tracking application, wherein execution of the object trackingapplication directs the processor to: receive a sequence of images,wherein at least one moving object is visible in relation to abackground in the sequence of images; perform image stabilizationrelative to a reference image in the sequence of images; estimate pixelbackground values based on an average of pixel values within the imagestabilized sequence of images; subtract the background pixel values frompixels in the image stabilized sequence of images to createbackground-subtracted pixels; compute sets of summed intensity valuesfor different per frame pixel offsets from the background-subtractedpixels, wherein a summed intensity value for a given per frame pixeloffset is computed by summing intensity values of thebackground-subtracted pixels from the image stabilized sequence ofimages determined using the given per frame pixel offset relative to apixel location in the reference image; identify summed intensity valuesfrom the sets of summed intensity values exceeding a threshold; clusterthe identified summed intensity values exceeding the thresholdcorresponding to single moving objects to form at least one summedintensity value cluster; and identify a location of at least one movingobject in the reference image based on the at least one summed intensityvalue cluster.